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^^ Abstract. The cluster variation method has been developed into a general theoretical 

^^ framework for treating short-range correlations in many-body systems after it was first proposed 

QQ by Kikuchi in 1951. On the numerical side, a message-passing approach called generalized belief 

T-H propagation (GBP) was proposed by Yedidia, Freeman and Weiss about a decade ago as a 

way of computing the minimal value of the cluster variational free energy and the marginal 

r ^ ' distributions of clusters of variables. However the GBP equations are often redundant, and it is 

|- ! quite a non-trivial task to make the GBP iteration converges to a fixed point. These drawbacks 

• hinder the application of the GBP approach to finite-dimensional frustrated and disordered 

f\ systems. 

I I In this work we report an alternative and simple derivation of the GBP equations 

starting from the partition function expression. Based on this derivation we propose a 

\ natural and systematic way of removing the redundance of the GBP equations. We apply 

'^ the simplified generalized belief propagation (SGBP) equations to the two-dimensional and 

. the three-dimensional ferromagnetic Ising model and Edwards-Anderson spin glass model. 

I The numerical results confirm that the SGBP message-passing approach is able to achieve 

w-\ satisfactory performance on these model systems. We also suggest that a subset of the SGBP 

' equations can be neglected in the numerical iteration process without affecting the final results. 

o 
m 

,^ 1. Introduction 

S^ Short loops are abundant in finite-dimensional ferromagnetic spin models and spin glass models. 

H They cause complicated and strong local correlations in the systems. The accumulation and 

propagation of these local correlations then lead to long-range correlations and the emergence 
of various collective behaviors. The cluster variation method (CVM) is a general theoretical 
framework for treating local correlations in many-body statistical systems. The original idea of 
the cluster variation method was conceived by Professor Ryoichi Kikuchi (1919-2003) in 1951 
m . Since then CVM has been applied to many different types of systems and has been further 
developed and generalized [21 [3l HI El E] . The basic idea of CVM is to decompose the entropy 
of the whole system into the residual entropy contributions of various clusters of spin variables. 
After this decomposition, the true entropy of the system is approximated by the sum of residual 
entropy contributions from a properly chosen subset of spin clusters, see [2] or section 2.2 of [7] 
for a brief introduction. 

The celebrated Bethe-Peierls tree approximation [U |9l [TOl \TT\ is an important limiting 
case of the cluster variation method. This tree approximation plays a central role in the 
mean-field theory of spin glasses [TU |T3], and it is also underlying the widely used belief 



propagation (BP) message-passing algorithm in information science [T3|. For various spin glass 
problems defined on finite-connectivity random graphs, due to the absence of short loops, 
the Bethe-Peierls approximation or its extended version (with the possibility of ergodicity 
breaking in the configuration space being considered) can give asymptotically exact results in the 
thermodynamic limit (see p[3] for a comprehensive review) . But the Bethe-Peierls approximation 
is inadequate in treating strong local correlations and therefore it performs poorly on finite- 
dimensional spin glass systems. 

Considering more local correlations beyond the level of Bethe-Peierls approximation is 
conceptually easy within the CVM framework, but for strongly disordered and frustrated spin 
glass systems this task is practically quite challenging. There are two major issues: (a) How 
to construct a suitable variational marginal probability distribution for each chosen cluster 
of spin variables? (b) How to efficiently minimize the total Kikuchi cluster variational free 
energy, a complicated function of a large set of parameters? Yedidia and co-authors proposed 
in p3] a particular way of constructing cluster marginal probability distribution functions, 
and then they suggested a generalized belief propagation (GBP) message-passing approach to 
minimize the resulting Kikuchi variational free energy. This GBP message-passing approach 
outperforms the BP message-passing approach considerably in terms of numerical precision, but 
its widespread applications on finite-dimensional spin glass systems are still hindered by two 
important drawbacks: first, the GBP equations are often redundant; and second, it is quite a 
non-trivial task to make the GBP iteration converges to a fixed point. 

In this work we understand the cluster marginal probability distribution functions of |15j 
from the viewpoint of the equilibrium partition function, and give an alternative and simple 
derivation of the GBP equations. Based on this derivation we propose a natural and systematic 
way of removing the redundance of the GBP equations. The resulting simplified generalized 
belief propagation (SGBP) equations are much more convenient for numerical implementation 
compared with the original GBP equations. We also point out that, for a given system, a 
subset of the SGBP equations can be safely ignored in the numerical iteration process. As 
redundance is minimized, the iteration process based on SGBP is much more easier to converge 
to a fixed point. We demonstrate the good performance of the SGBP equations by applying 
these equations to the two-dimensional (2D) and the three-dimensional (3D) ferromagnetic Ising 
model and Edwards- Anderson spin glass model. 

The next section defines the general model system. The GBP equations are derived in 
section [3] starting from the partition function. These equations are then simplified in section |4j 
Some numerical results obtained by the SGBP equations are discussed in section [5| We conclude 
this work in section [HI 

2. The general model system 

We first define the general model system and briefly describe the region graph concept. A 
convenient expression for the partition function of the system is given. 

2.1. Energy 

Consider a system with A^ vertices and M two- or many-body interactions among the vertices. 
The indices of the vertices are denoted as i,j, /c, . . . G [1, N] and those of the interaction terms 
as a,b,c, . . . G [1, M]. Each vertex i has a state Xj which, for notational simplicity, is assumed 
to be a discrete scalar variable. A microscopic configuration of the system is denoted as x, with 
x = {xi,...,Xj,..., 2; at}. The energy function has the following general form 
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Figure 1. (a) Part of the factor graph for the Edwards- Anderson model on a square lattice 
(with periodic boundary condition on each side). Each small circle (such as i) denotes a vertex 
(lattice site); each small square (such as a) denotes a spin coupling interaction. An edge (i,a) 
between a circle i and a square a means that vertex i participates in interaction a. (b) Part 
of a redundant region graph for the factor graph of (a). Each square region (highlighted in 
cyan, such as a and (3) contains four vertices and four interactions; each rod region (highlighted 
in green, such as a and b) contains two vertices and one interaction; and each vertex region 
(highlighted in red, such as i and j) contains only one vertex. A directed edge points from a 
parent region to a child region. Each vertex appears in 9 regions (see the example of vertex i); 
and each interaction appears in 3 regions (see the example of interaction a between vertex i and 
j). The counting number for a square region, a rod region and a vertex region is c = 1, c = —1 
and c = 1, respectively. 



In the above expression, Ei{xi) and Ea{xQ^) are, respectively, the self-energy of vertex i and 
the energy of interaction a; da denotes the set of vertices involved in interaction a, and 
^da = {^^iK ^ ^'^l is ^ microscopic sub-configuration for this particular set of vertices. 

To give a concrete example of the general model system (IT]), let us mention the Edwards- 
Anderson (EA) spin glass system on a finite-dimensional regular lattice [16J. The vertices are 
then the lattice sites, each of them having a binary spin state (xj G {—1, +!})• There is a spin 
coupling interaction between each pair of nearest neighboring lattice sites. The energy of a given 
spin configuration is 



N 



H{x) 






(2) 



where h^ is the external field on vertex (lattice site) i, and (i, j) denotes a pair of nearest- 
neighboring vertices i and j, with Jij being the coupling constant between them. 

A conventional graphical representation for the general model system (IT]) is a bipartite graph 
of A^ small circles (representing the vertices) and M small squares (representing the interactions) . 
Such a bipartite graph is referred to as a factor graph in the literature [T7|. Each edge in the 
factor graph is between a small circle and a small square. If and only if a vertex i is involved in 
an interaction a, then there will be an edge connecting the corresponding small circle and small 
square in the factor graph. As an example, figure |l(a) shows part of the factor graph for the 
EA model (|2]) on a periodic square lattice. 



2.2. Partition function and free energy 

The partition function of system (IT]) has the sum-product form 



Af M 



Z{P) = J]e-^^fe) =Y.\{U^^)\{Mxaa) ■ (3) 

X X i=l a=l 

The inverse temperature /3 = ^r^, with T being the temperature (we shall set Boltzmann's 
constant A:^ = 1 in the following discussions). The functions tpi{xi) and ipainUda) ^^6, respectively, 
the Boltzmann factor for the self-energy Ei and the interaction energy Ea, 

i;i{xi) EE e-/^^»(-') , ^l^a{x9,) = e-^^"fes«) . (4) 

The equilibrium free energy i^(/3) is related with the partition function Z{I5) as 

F(/3) = -ilnZ(/3). (5) 

Knowing the free energy F as a function of inverse temperature /3 and (if necessary) other 
environmental control parameters, we can then calculate all the other thermodynamic quantities 
such as the mean energy, the entropy, the mean value of Xi for each vertex i. The free energy 
and the partition function therefore have fundamental importance in equilibrium statistical 
mechanics. But exactly computing the partition function (and the free energy) is an impossible 
task generically. Many numerical schemes have been developed over the years to obtain good 
approximate values for 2'(/3) [1]. 

A well-known theoretical framework for performing approximation is Kikuchi's cluster 
variation method [H El IH [5]. However, the variational problem of minimizing the Kikuchi 
cluster free energy is rather difficult to solve, especially for spin glass systems with quenched 
random parameters. Yedidia, Freeman, and Weiss jT^] demonstrated that the minimal points 
of the Kikuchi variational free energy correspond to fixed points of a set of self-consistent 
generalized belief propagation (GBP) equations. Minimisation of the Kikuchi cluster free energy 
was therefore turned into the problem of constructing a fixed point for the GBP equations, 
which can be achieved through an iterative message-passing process on a region graph (see 
next subsection). Unfortunately, the GBP iterative process is still computationally demanding, 
especially when the underlying region graph are redundant. 

In this work we will give an alterative and simple derivation of the GBP equations starting 
from the partition function ([3|. An advantage of this derivation is that it suggests a natural 
way of simplifying the GBP equations on redundant region graphs. 

2.3. Region graph representation 

The basic motivation of the region graph concept is to facilitate treatments of local correlations 
through distributing vertices into different overlapping groups, with each group containing a 
subset of vertices that are believed to be strongly correlated [H EJ [15] . 

A region graph R\s a, graph composed of regions and directed edges between pairs of regions 
|15| . The regions are generally denoted by Greek symbols. Each region 7 contains a subset of 
the vertices and a subset of the self-energies and interaction energies. If there is a directed edge 
from a region /i to another region i/, denoted as /i — )• i/, then u must be contained in /i (namely 
all the vertices and all the energy contained in v must also be contained in ^). When the edge 
// — >• J/ is present, we say that /x is a parent of u and u a child of ^i. If there is a directed path from 
a region a to another region 7, we say that a is an ancestor of 7 and 7 a descendant of a, and 
denote this ancestor-descendant relationship by a > 7 and 7 < a. The notation a > 7 (a < 7) 



is understood as either a and 7 are identical to each other or a is an ancestor (descendant) of 

7- 

Figure 1(b) [shows part of a region graph for the EA model of figure 1(a) There are three 



types of regions. Each square region contains four vertices and four interactions, and it is parent 
of four rod regions; each rod region contains two vertices and one interaction, and it is parent 
of two vertex regions; each vertex region contains a single vertex. In this particular example, 
for notational simplicity, each rod region is denoted by the index of the single interaction in it, 
and each vertex region is denoted by the index of the single vertex in it. 

Each region 7 is assigned a counting number c^ [2|, which is constructed recursively by 

C7 = 1 - ^ Ca . (6) 

{a : a>7} 

If a region a has no directed edges pointing to it, its counting number is c^ = 1. For a region fj, 
with ancestors, it is obvious from the construction ([6J) that c^ + J2'y>n.^'y = ^■ 

The vertices and energy terms of system M are clustered into various regions of R. This 
clustering, however, is not exclusive. A vertex and an energy term may be assigned to more 
than one region. To ensure that each energy term contributes only one Boltzmann factor to the 
partition function ([3]), the region graph R is required to satisfy the following two constraints 
[15]: (1) For any vertex i, the induced subgraph Ri (i.e., the region subgraph formed by all the 
regions containing i and all the directed edges between pairs of these regions) is connected, and 
the sum of counting numbers within Ri is unity: 

E ^7 = 1 ; (7) 

and (2) for any interaction a, the induced subgraph Ra (i.e., the region subgraph formed by all 
the regions containing a and all the directed edges between pairs of these regions) is connected, 
and the sum of counting numbers within Ra is unity: 

E ^7 = 1 • (8) 

-y&Ra 

Because of Q and ([s]), the partition function can be expressed as 






(9) 
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In our earlier work |181 [7], the expression ([9]) was the starting point for performing partition 
function expansion on the region graph R. 

For each region a of a given region graph R, let us denote by /^ = {7 ^ 7 < 0} the set formed 
by region a and all its descendants, by Aa = {7 : 7 > a} the set formed by all the regions 
ancestral to region a, and by B^ the set that contains all the regions not belonging to set /„ 
but parental to at least one region of set /« [7]. We refer to the set I^ as the interior of region 
a and set Ba as the boundary of region a. As some examples, we show in figure [2] the interiors 
and boundaries of the square region a, the rod region a and the vertex region i of the region 



graph of figure 1(b) 



A region graph R is referred to as non-redundant if the region subgraph Ri induced by any 
vertex i is a tree (containing no loops), otherwise R is referred to as redundant |18t [7]. In a 
non-redundant region graph there is only one directed path from a region a to a descendant 
region 7, while in a redundant region graph there may exist multiple directed paths from an 



ancestral region a to a descendant region 7. The region graph shown in figure 1(b) is redundant, 
since there are two directed paths {a ^ a ^ i and a — )• d — )• z) from the square region a to the 
vertex region i. 
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Figure 2. The interior and boundary for the square region a (a), the rod region a (b), and the 



vertex region i (c) of figure 1(b) The interior set Ij of any region 7 is marked by a pink-colored 



domain, while the boundary set B^ of region 7 is marked by a yellow-colored domain. 



3. Generalized belief propagation (GBP) equations 

We give an alternative derivation of the generalized belief propagation (GBP) equations |15j in 
this section. Let us introduce on each directed edge /x — t- i^ of the region graph R an arbitrary 
probability distribution function m^_>.,y(x^), with the only constraints that this function is 
positive and is properly normalized, J2x '^'^t^.-^^i^u) — 1- The function mfj_^y{x^) is a probability 
measure on the microscopic states x^ = {xi\i ^ v} oi region v. This measure is applied on v by 
the parent region ^. The probability m^^u{x^) can be regarded as a message from the parent 
region /i to the child region v concerning the microscopic sub-configuration x^. 
We observe that, for each directed edge ^ ^ u, 






1-1 = 0. 



This identity guarantees that the partition function ([9]) can be expressed as 
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For each region a let us define a Boltzmann factor Za as 



^" = z] n ^*(^*) n ^«(^9a) 



n 



m 



'fJL^U\X_V 



x^ iea aeo {/^■^i' ■ /J.&Ba,iy€la} 

Then the partition function can be re-written as 

Z(/3) = ZoX^ll u^o^ix^r- , 

X a&R 

where -^0 = H ^a' ) ^"^^ ^^^ weight oja is defined as 
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(10) 



:ii) 



(12) 
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(14) 



By exploiting the expression (13) we can write the free energy F(/3) as F(/3) = Fq + AF, with 
„ 1 " 
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(15) 



and correction contribution AF = —4 ln[^^ Yia&R^cii^aT"]- 

Let us assume that AF can be safely neglected in comparison with Fq. Then the free energy 
can be approximated as F{f3) ^ Fq. A rigorous justification of this approximation is absent, 
but it was shown in jl8| [7] that, AF is the sum of correction contributions from looped region 
sub-graphs at least in the cases of non-redundant region graphs. We hope to return to this 
question in a future work. 



The expression (15) for Fq still contains all the auxiliary probability distribution functions. 



Naturally we require the free energy Fq to be stationary with respect to the chosen set of 
probability functions {m^_>.iy}. In other words, the first derivative of Fq with respect to any 



m 



■fi^u 



should be zero. 
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This condition is satisfied if we require the auxiliary functions to be chosen in such a way that, 
for each directed edge // — t- i/ of the region graph G, 






UJu{x^) ■ 



(17) 



Equation (|17|) ensures the consistency of the two marginal probability distributions uj 



and Wiy(Xy) of each parent-child pair {^,1^). Equation (14) and Eq. (17) lead to the following 



fj.\i^fii 



generalized belief-propagation equation on each directed edge /x — )• z/ of the region graph R: 



^ y H ^i^^i^ n '^biXgt,) 



^A^vj^fj-y 
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II m^^riXr) , (18) 



where C is an adjustable constant to ensure that m^^jj{x^) is properly normalized. The set of 
GBP equations (18) was first derived in pSj through a different approach. 

If the region graph R is non-redundant, then for each directed edge /i — )• i/, i?i/ n /^ = {/x} 
and {a — )• 7 : a G i?i^ n /^, 7 G Ii,} = {/.i — )■ v}. Then equation ([T8|) is simplified as 



m„ 



'^ X] n ^j'(^j) n '^b{XQb) 
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(19) 
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It can be shown that equation ( 19 ) is equivalent to the region graph belief propagation (rgBP) 



equation of [IHl \7\ obtained through partition function expansion. 

If the region graph R is redundant, then there are multiple directed paths between some pairs 
of regions. As a consequence, the consistency conditions ( [l7[ ) on some of the directed edges must 
be redundant. For example, because there are two directed paths a — >■ a — )• z and a — >• d — >■ i 



from the square region a to the vertex region i in figure 2(b) then if the consistency conditions on 
the edges a — )• a, a — )• d and a ^ i are all satisfied, the consistency condition on the edge d ^ i 



is satisfied automatically. For a redundant region graph, some of the consistency conditions (17) 



can therefore be dropped without affecting the final theoretical results. Consequently, some of 



the GBP equations ( 18 ) do not need to be considered. 



4. Simplified GBP (SGBP) on a redundant region graph 

In this paper we are interested in the case of the region graph R being redundant. For example, 



figure 1(b) shows a redundant region graph for the 2D Edwards- Anderson model on a square 



lattice with period boundary conditions. In this region graph, the region subgraph Ri induced 
by each vertex i is not a tree. It contains nine regions and twelve directed edges, and loops exist 
in Ri at the level of regions. 

According to the definition (12), each boundary edge (/x — )• z^) of region a contributes a 
product term m^^^y{xjj) to the Boltzmann factor Za- As an example, consider the rod region a 
of figure 2(b)[ Its Boltzmann factor Za is 



2^a = ^ 1piiXi)lpj{xj)llJa{xi,Xj)ma-^a{Xi,Xj)mi3_^a{xi,Xj)mh^i{xi)mf^j{Xj) 

xmd^i{xi)mg^i{xi)mb^j{xj)me-^j{xj) . (20) 

Notice that the square region a and its two child rod regions b and d all send a message to 
region o or its descendants, and these three messages are assumed to be mutually independent 
in the probability product form of equation (20). Similarly, the square region /3 and its 
two child rod regions e and g contribute a product term of three probability messages to 
Za- However, because the rod regions b and d are children of region a, the probability 
message ma^a{xi,Xj) already contains the effects of region b and d to region a. In other 
words, the probability distributions mii^i{xi) and mh^j{xj) very likely are strongly related to 
the probability distribution ma^a{xi-,Xj). Similarly, we expect the probability distributions 



i{xi) and me^jixj] 



to be strongly related to the probability distribution mp^a{xi,Xj). 



All such possibly strong dependence effects among the input probability distributions are 
completely neglected in (20) and in the more general expression (12). In this sense the 



redundance in the region graph structure causes redundance in the Boltzmann factor expressions 



(12), which in turn contributes at least part of the redundance in the set of GBP equations 



(18) and increases the complexity of numerical computations. The issue of removing the GBP 



redundance has been discussed by several recent studies 13 El |T9l [201 EI] • 

We now propose a new way of removing redundance in the GBP equations. The basic idea is 
very simple: for a region v in the boundary set Ba of region a, the input probability messages 
from V to the set la should be removed as many as possible if an ancestor of v also belongs to 
the boundary set Ba- Let us first discuss the ideal situation, namely the following equality is 
valid on each directed edge ^ ^ v oi the region graph R- 



E 







(21) 



{a : ti&Ba,u&Ia,Ai^r\Ba=d} 



where An is the ancestor set of region /x as defined in section 2.3 The summation on the left' 



hand side of the above expression is over all the regions a satisfying the following conditions: 

(1) the boundary set Ba contains region /i but not any of the ancestors of fi (A^ n Ba = 0); and 

(2) the interior set la contains region i^. As a simple example, let us mention that identity (21) 



is satisfied on each directed edge of the redundant region graph of figure 1 (b) This particular 
region graph is therefore redundant and 'ideal'. 



If (21 ) holds on each directed edge /x 



then the partition function expression (11) can be 



simplified as 
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We can then define a simplified Boltzmann factor for region a as 
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A simplified Boltzmann weight for region a is defined correspondingly: 
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(24) 



For the ideal redundant region graph of figure 1(b) , the Boltzmann factor Za of the rod region 
a is readily written down as (see figure 2(b)[ ) 

Za = ^ ^l^i{Xi)%l}j{Xj)'lpa{Xi,Xj)mo,-,a{Xi,Xj)mi3^a{Xi,Xj)mh^i{xi)mf^j{xj) , (25) 



which is much simpler than the expression (20). 
The partition function is then expressed as 



^a[Xa) " , 



(26) 



where -^o = 11 ^a" • A new approximate free energy Fq is defined as 
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(27) 

Requiring Fq to be stationary with respect to the set of auxiliary probability distributions 
{m^^u{Xj^)}, the following set of simplified consistency conditions are obtained: 






V(/i ^ I/) e i? . 



(28) 



This consistency condition leads to the following simplified generalized belief propagation 
(SGBP) equation on each directed edge /i — )• z/: 

{a-s.7 : a£B„r\I^,-y&I^,Aar\B^=<l)} 

^ E n ^j(^i) n '^b(^db) n rn,^r(^.) , (29) 

where C is again an adjustable normalisation constant. 

For a given spin glass model M, it may be a relatively easy task to construct a redundant 
region graph that has the property ^T\\ on each of its directed edges. The set of SGBP equations 
(29) can then be iterated on such an 'ideal' redundant region graph i?, and the approximate free 
energy Fq can be computed accordingly. 

To be complete, let us also briefiy discuss the 'non-ideal' case in which the identity (21 ) holds 
on some but not all of the directed edges. A simple solution would be to divide the directed 
edges into two classes (say C/ and C^v), one (C/) with the property (21) and the other one 



(Cat) with only the weaker property (10). For an edge (/i — )• z^) S C/, the probabihty message 
'm^^iy{x^) is received by all the regions a in the set {a : /i G Ba-, v G la^A^^ n B^ = 0}. While 
for an edge (r/ — )• r) S Cat, we require the probability message m^^r(a^T) ^o be received by all 
the regions in a properly constructed non-empty subset of the set {7 : r/ G B^^t S I-^} (this 
constructed subset has the property that the sum of counting numbers of its regions is equal to 
zero). Following the theoretical approach of this section we can then obtain a modified set of 
SGBP equations for the non-ideal region graph K. 

As there are multiple directed paths between some pairs of regions in the redundant region 



graph i?, the simplified consistency conditions (28) on some of the directed edges must be 
redundant (see the discussion at the last paragraph of section ^. Therefore some of these 
simplified consistency conditions and the corresponding SGBP equations do not need to be 
considered in the actual numerical iteration process. 

5. Applications to the Ising and the Edwards-Anderson model 

It is helpful to complement the theoretical discussions of the preceding section with some 
applications. We now apply the SGBP equations to the ferromagnetic Ising model and the 
Edwards- Anderson spin glass model (pi) on a square lattice or a cubic lattice. Periodic boundary 
condition is assumed on each dimension of the lattice systems. For the Ising model, each coupling 
constant Jij = J, while for the EA model Jjj is assigned a value -|-J or —J independently and 
uniformly at random and then is fixed in time (we shall set J = 1 in the following discussions). 



For the ideal redundant region graph (later referred to as ^20"^) of figure 1(b) , the simplified 
Boltzmann weights of a square region a, a rod region a and a vertex region i can be easily 
written down as (see figure [2]) 

W (x- X- Xh Xl) ex gl^ih'^Xi+h'^Xj+hlxk+hfxi+JijXiXj+JjjcXjXk+JklXkXl+JliXiXi) 

xmi3^a{xi,Xj)m^^h{xj,Xk)m^^c{xk,xi)m^^d{xuXi) , (30a) 

OJa{x^,Xj) OC e'^^^°^'''^^°^''^^'^'^'''''^'^ma^aixi,Xj)mf3^a{Xi,Xj)mh^i{Xi)mf^j{xj) , (306) 

Ui{xi) ex e'^^^''"ma^i{xi)mh^i{xi)md^i{xi)'mg^i{xi) . (30c) 

The SGBP equations on the directed edges a ^ a and a ^ i are, respectively 

ma-,aiXi,Xj)mh^i{x,)mf^j{Xj) OC ^ ^mi^k+h'^^i+J.kX.Xk+JMXkXi+JuXtx,) 

Xk,Xl 

xm^^b{xj,Xk)mf,^cixk,xi)mj^d{xi,Xi) , (31a) 
ma^i{xi)md^i{xi)mg^i{xi) ex ^e^^^i''^~^-^"'''''''>ma^a{xi,Xj)mi3^a.{xi,Xj)mf_^j{xj) . (316) 

To include more local correlations in the SGBP equations, we also consider another ideal 
redundant region graph (referred to as R^d^, see figure p| for the 2D system (pi). This region 
graph has the same topology as R2d^- The only difference is that each vertex region now 
becomes a plaquette region of 2 x 2 vertices, and each square region now contains 4x4 vertices. 



The SGBP equations for this region graph are similar to equations (31a) and (316). 

For the cubic lattice systems we consider a simple 3D extension of the region graph of 
figure 1(b) This extended region graph (later referred to as R^^^) has four types of regions: 



cube regions, surface regions, rod regions, and vertex regions. Each 2x2x2 cubic cell of the 
3D lattice forms a cube region of the region graph, it is parental to six surface regions, and its 
counting number is c = 1; each 2x2 plaquette of the lattice forms a surface region, it is parental 
to four rod regions and has counting number c = — 1; each pair of nearest neighboring vertices 
of the lattice forms a rod region (it is parental to two vertex regions, and its counting number 




Figure 3. The local structure of another redundant region 
graph for the 2D model of figure 1(a) Each square region 



(cyan color) contains 4x4 vertices and is parental to four rod 
regions, its counting number is c = 1; each rod region (green 
color) contains 2x4 vertices and is parental to two plaquette 
regions, its counting number is c = —1; each plaquette region 
(pink color) contains 2x2 vertices, its counting number is 
c = 1. For simplicity we do not show the small squares of 



figure 1(a) here but use a small edge between two vertices to 



indicate a coupling interaction. 



is c = 1); and each vertex region contains only one single vertex, with counting number c = — 1. 
The SGBP equations for such a 3D region graph are much more simplified in comparison with 
the original GBP equations. 

We are mainly interested in the spontaneous emergence of collective behaviors, therefore we 
set the external field on each vertex to be zero {h^ = for all the vertices i). 



5.1. The 2D Ising model 

First we consider the simplest case, namely the ferromagnetic Ising model on an infinite square 
lattice with periodic boundary conditions. As the system has no disorder the SGBP equations 
can be easily solved at all temperatures. In figure |4] we compare the results obtained by the 
SGBP equations with the exact results of Onsager |22j and with the results obtained through 
the region graph belief propagation (rgBP) equations (see [U [18j for details). 

For the SGBP equations on a given region graph, we perform linear stability analysis to 
determine precisely the instability point of the paramagnetic solution, similar to that performed 
in ^. The predicted transition inverse temperatures by the SGBP equations using the region 
graph -R2D^ ^'^^ ^2D^ ^^^' respectively, /3c ~ 0.4126 and /3c k, 0.429. These values are rather 
close to the exactly known critical inverse temperature value of p^^""^ ~ 0.4407 [2^, they are 
much improved over the predicted value of /3c ~ 0.3466 by the conventional BP method and the 
predicted values of /3c ~ 0.3765 and /3c ~ 0.401 by the rgBP equations on two non-redundant 
region graphs [7j. 

The SGBP equations also give better predictions on the free energy density, the mean energy 
density, the entropy density, and the mean magnetisation of the system, see figure |4j Using the 
region graph R^^^, the predicted values of the SGBP equations on the free energy density, energy 
density, and entropy density become very close to the exactly known results. The prediction 
power of the SGBP equations will be further improved if we consider more local correlations by 
enlarging each region of figure [3] while keeping unchanged the topology of this region graph. 

5.2. The 2D Edwards- Anderson model 

Because of the existence of multiple directed paths between some pairs of regions in the 
redundant region graph i?2B^ °^ -^2D^' ^^^ SGBP equations on some of the directed edges 
are redundant and can be dropped from the numerical iteration process (see discussion in the 
last paragraph of section El). If instead the SGBP equations on all the directed edges of the 
region graph are used in the iteration process, the paramagnetic solution of the SGBP equations 
will be unstable even at /3 = (T = oo). 

In our numerical implementations for the 2D Edwards- Anderson model we still keep the SGBP 
equations on all the directed edges but add a damping effect to the SGBP equations to facilitate 
the convergence. We notice that the paramagnetic solution of the SGBP equations on the 
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Figure 4. Results on the 2D ferromagnetic Ising model obtained by the SGBP equations, the 
rgBP equations of 0, and the conventional BP equation: (a) free energy density; (b) mean 
energy density; (c) entropy density; (d) magnetisation (mean spin value of a single vertex). The 
solid lines denote the exact results of [:22j. The parameter n of the SGBP equations and the 
rgBP equations is the number of vertices on each side of the maximal square region of the region 
graph. For the SGBP curves, n = 2 and n = 4 correspond to the region graph of figure |l(b) 
and figure [Sj respectively. The region graphs at n = 2 and n = 4 for the rgBP equations are 
described in |7]. 



region graphs R!^^'^ and R^^* fail to be stable at low temperatures even for optimally adjusted 
magnitude of the damping effect. However, the it J EA model on 2D square lattice is believed 
to have no finite-temperature spin glass phase [23] • Therefore only the paramagnetic solution of 
the SGBP equations are qualitatively correct, even if it is unstable at low temperatures. 

Since the free energy of a given instance of the EA model on a periodic square lattice with 
side length L can be computed exactly in polynomial time [25], we can compare the free energy 
Fq of the paramagnetic solution of the SGBP equations with the exact value, see figure [5J 
The SGBP results on the region graph i?2D^ ^^^ comparable to the results obtained by the 
tensor renormalisation group (TRG) method [2ii| with cutoff parameter D = 8 but are worse 
than the TRG results with cutoff parameter D = 16. We expect that the performance of the 
SGBP equations will be further improved by enlarging the side length n of the square regions. 
An advantage of the SGBP message passing approach is that it can simultaneously give the 
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Figure 5. Difference A In Z between the 
exactly calculated value of In Z and the 
value of In Z obtained by BP, rgBP with 
n = 2, SGBP with n = 2 and n = 4, and 
tensor renornialisation group (TRG) with 
cutoff parameter D = 8 and D = 16. All 
the numerical calculations are performed 
on a single instance of the 2D EA model 
with side length L = 64 and periodic 
boundary conditions (the total number of 
vertices is A^ = 64^). 



5.3. The 3D Ising model and Edwards- Anderson model 

The structure of a simple redundant region graph R^^^ for the 3D Ising model and EA model 
is described at the beginning of section [5] This region graph is formed by cube regions, surface 
regions, rod regions and vertex regions. Each cube region contains nxnxn vertices with n = 2. 
Linear stability analysis of the SGBP equations predicts that the ferromagnetic transition occurs 
at the critical inverse temperature /3c ~ 0.2183, which is very close to the value of /3c ~ 0.2217 
obtained by Monte Carlo simulations [27] and the higher-order tensor renornialisation group 
(TRG) method ^8j. As a comparison, the critical inverse temperature predicted by the BP 
approximation is (3c ~ 0.2027. 

The magnetisation values predicted by the BP and SGBP equations are compared with the 
Monte Carlo simulation results of [27] in figure pj It is evident that the performance of SGBP 
improves considerably over that of BP. 

We also perform linear stability analysis on the paramagnetic solution of the SGBP equations 
for the 3D EA model Q. The stability threshold /3c of the SGBP paramagnetic solution on 
the region graph R^^'^ depends on the particular disorder instance. For each cubic lattice of 
side length L under the periodic boundary condition, we generate 32 independent realisations of 
the coupling constants {Jij} and then obtain the value of /3c for each disorder realisation. The 
relationship between the mean value of /3c and lattice size L is shown in figure [7| The mean 
/3c value decreases with lattice side length L and appears to approach the value of /3c ~ 0.505 
at L — )• oo. As a comparison, Monte Carlo simulations predicted that the spin glass phase 
transition occurs at f3c ~ 0.893 (see |29j and references therein). We believe that if we enlarge 
the side length n of the maximal cube region of the used region graph, the critical value (3c at 
L = oo will increase and be much more closer to the value predicted by Monte Carlo simulations. 



6. Conclusion 

In summary, we presented an alternative way of deriving a set of generalized belief propagation 
(GBP) equations for a general lattice statistical physical system. Based on this derivation we 
proposed a way of simplifying the GBP equations. We also pointed out that, due to the existence 
of redundance in the region graph, some of the simplified generalized belief propagation (SGBP) 
equations can be ignored in the numerical iteration process. We applied the set of SGBP 
equations to the two-dimensional and three-dimensional Ising model and Edwards-Anderson 
model. The numerical results confirmed that the SGBP message-passing approach significantly 
outperforms the conventional BP approach in treating short-range correlations. Hopefully 
this work will stimulate further developments of the SGBP message-passing approach and the 
applications of this approach to finite-dimensional disordered systems and finite-dimensional 
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optimisation problems. 

The three-dimensional Edwards-Anderson model ([2]) is in the spin glass phase at low 
temperatures, with broken ergodicity. To describe the low-temperature spin glass property 
of the EA model, the effects of ergodicity-breaking need to explicitly considered in the SGBP 
equations. This issue remains to be solved. 
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